Skip to content

Redis support via Jedis client#212

Merged
tylerbenson merged 10 commits into
DataDog:masterfrom
gihad:jedis
Feb 23, 2018
Merged

Redis support via Jedis client#212
tylerbenson merged 10 commits into
DataDog:masterfrom
gihad:jedis

Conversation

@gihad
Copy link
Copy Markdown
Contributor

@gihad gihad commented Feb 5, 2018

This adds Jedis 2.9.0 client support

Some improvements that could be made:

  • Sending the arguments from the redis commands to the datadog servers similar to the Ruby redis integration (note: Newrelic and Glowroot java APM also doesn't have the arguments). I haven't found a way to do this without some implications on performance.

  • As seen on the last screenshot there seems to be an issue reporting really short spans

image

image

image

Sometimes when the time spent in redis is too low it's not reported accurately (e.g.: 1ns):
image

At first I thought maybe the method being instrumented wasn't doing any io which would explain the very low 1ns recorded time for the redis span, but after looking at the Jedis code it looks like IO is included in sendCommand. Also, other APM tools instrument the same method I did here: e.g.: https://github.com/glowroot/glowroot/blob/master/agent/plugins/redis-plugin/src/main/resources/META-INF/glowroot.plugin.json
Finally, I see similar numbers in New Relic.

@palazzem palazzem added inst: others All other instrumentations tag: community Community contribution labels Feb 5, 2018
@palazzem palazzem added this to the 0.4.0 milestone Feb 5, 2018
@tylerbenson
Copy link
Copy Markdown
Contributor

Hey @gihad,
Nice work on this PR. I wanted to see if you'd be comfortable adding some automated tests. (Perhaps using an embedded redis server so the real driver is exercised.) We have a slick system that allows the tests to be run with the instrumentation automatically applied by using AgentTestRunner. Please take a look at CassandraClientTest and KafkaTest for examples of the kind of verification I like to see.

If you're not comfortable with that, or you don't have more time to commit let me know.
Thanks!

span.setTag(DDTags.RESOURCE_NAME, command.name());
span.setTag(DDTags.SERVICE_NAME, "redis");
span.setTag(DDTags.SPAN_TYPE, "redis");
span.setTag("span.origin.type", command.getClass().getName());
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the command class is always the same, then this tag is probably not useful. I added that in other instrumentation because that can be an important tool for debugging.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense. Will remove.

Tags.SPAN_KIND.set(span, Tags.SPAN_KIND_CLIENT);
Tags.COMPONENT.set(span, "redis-command");

span.setTag(DDTags.RESOURCE_NAME, command.name());
Copy link
Copy Markdown
Contributor

@tylerbenson tylerbenson Feb 5, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What are some examples of command.name()? Does this have the right level of cardinality to be useful?

It would be nice if we could name the resource off of more than just the command name, but perhaps the full command (assuming it doesn't change a lot). Though based on your PR description, it sounds like this isn't possible.

Copy link
Copy Markdown
Contributor Author

@gihad gihad Feb 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Examples of commands are SET, GET, ZRANK and all the other redis commands described here: https://github.com/xetorthio/jedis/blob/2.9/src/main/java/redis/clients/jedis/Protocol.java#L244
I think even at this granularity it is useful

Having the command name + the arguments to the command instead is something I listed in the PR description as a possible improvement, however there are some things to consider:

The datadog ruby APM does that (SETEX command + arguments):

image

The NewRelic and Glowroot Java APM for redis only list the Commands like this one. I suspect it has something to do with how the Jedis code is laid out. There are 3 methods:

Method 1:

 protected Connection sendCommand(final Command cmd, final String... args) {
    final byte[][] bargs = new byte[args.length][];
    for (int i = 0; i < args.length; i++) {
      bargs[i] = SafeEncoder.encode(args[i]);
    }
    return sendCommand(cmd, bargs);
  }

Method 2:

protected Connection sendCommand(final Command cmd) {
   // declared above private static final byte[][] EMPTY_ARGS = new byte[0][];
    return sendCommand(cmd, EMPTY_ARGS);
  }

Method 3:

protected Connection sendCommand(final Command cmd, final byte[]... args) {
...
}

As you can see methods 1 and 2 are just wrappers for method 3. In my instrumentation I'm just monitoring method 3 and not converting the byte[] args back to String. Here are some options:

  • Convert the byte[] back to String to send the arguments along the command. I decided against this for performance reasons.

  • Only monitor Method 1 which has sendCommand(final Command cmd, final String... args). I decided against this because we would lose instrumentation for redis commands that don't have arguments since they use method 2.

  • Create separate Advices to monitor method 1 and 2 and not monitor method 3. I tried this but the code ended up convoluted and I didn't get to work. Didn't spend much time trying it tho.

Having this info do you have any suggestions on how to proceed or do you think we should just leave it the way it is (just the command name)?

Copy link
Copy Markdown
Contributor

@tylerbenson tylerbenson Feb 7, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am fine having resource name be just the command name. That is something we can iterate on in the future.

Later, if we decide we want to capture the arguments, we can do what the ruby client does and set the full command + arguments in the redis.raw_command tag (ex). The ruby client does have an additional level of quantization, but I don't quite understand it so I don't expect you to implement that.

import redis.clients.jedis.Protocol.Command;

@AutoService(Instrumenter.class)
public final class JedisInstrumentation implements Instrumenter {
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We just merged #213 which adds a new abstract parent for instrumentation to use. Please rebase from master and change your new instrumentation to instead extends Instrumenter.Configurable, add a constructor that defines some useful names for this instrumentation and also add an override similar to what I did for kafka so that it is disabled by default. This will allow folks to test it for a while if they desire. At a later release we can then remove that and enable it by default.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, will do.

@gihad
Copy link
Copy Markdown
Contributor Author

gihad commented Feb 9, 2018

@tylerbenson I can address all the comments here including the embedded redis tests but I won't be able to get to it soon. Whatever time I have available for datadog work I need to spend at getting other aspects of APM working (e.g.: currently errors wrapped by spring boot don't show up in datadog).

@gihad
Copy link
Copy Markdown
Contributor Author

gihad commented Feb 18, 2018

@tylerbenson finally got around to working on this. All comments have been addressed.

Copy link
Copy Markdown
Contributor

@tylerbenson tylerbenson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job and thanks for the contribution.

If you can tell me what the earliest version this supports, I can make the below recommended changes after merging (unless you want to do it yourself).

apply from: "${rootDir}/gradle/java.gradle"

dependencies {
compileOnly group: 'redis.clients', name: 'jedis', version: '2.9.0'
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry... one last thing and I think we're good. Generally, the convention is that the version here is the earliest version the instrumentation will work with. Will this work with anything before 2.9?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tylerbenson I just did a scan on all the tags available in the Jedis Repo (e.g.: https://github.com/xetorthio/jedis/blob/jedis-1.4.0/src/main/java/redis/clients/jedis/Connection.java)

My findings is that the first version that will be supported by this is Jedis 1.4.0 and it will work all the way to 2.10. However the current class in Master has changed the method and it will no longer work in their next release so we should probably cap the support at 2.10

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since it hasn't been released yet, our tooling can't really exclude it well. Let's just rename the project to jedis-1.4 and add some classes to identify it differently from pre-1.4. Would you like to do that or would you rather I do it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tylerbenson Just want to make sure you are aware that 1.4.0 is a really old version from Nov 15, 2010 and also one of the first versions (https://github.com/xetorthio/jedis/releases?after=jedis-1.5.0-RC1). Before 1.4.0 all the way down to 1.0.0 the method to instrument has this signature: https://github.com/xetorthio/jedis/blob/jedis-1.3.1/src/main/java/redis/clients/jedis/Connection.java#L52

I don't think I will get to it anytime soon so I rather you do it. Thanks.

Also, we will probably have to instrument another class after the next release of Jedis comes out (code in master now) anyways so we should plan for multiple instrumentations either way.

@Override
public AgentBuilder apply(final AgentBuilder agentBuilder) {
return agentBuilder
.type(not(isInterface()).and(hasSuperType(named("redis.clients.jedis.Connection"))))
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way we restrict the instrumentation from getting added to a version that isn't supported is by using this kind of thing: https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/instrumentation/aws-sdk/src/main/java/datadog/trace/instrumentation/aws/AWSClientInstrumentation.java#L34-L39.

Those classes are verified using a plugin: https://github.com/DataDog/dd-trace-java/blob/master/dd-java-agent/instrumentation/aws-sdk/aws-sdk.gradle#L5-L14

Basically those classes are unique to the versions of the library it supports.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I switched to the protocol instead of the connection since earlier versions of the no-arg sendCommand didn't delegate the same way. Do you see any problems with this @gihad?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only thing I see is that the sendCommand method from Connection does more work, not sure what's more accurate to instrument in this case. I'm good either way.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@realark I forget... is this needed for spring boot executable jar type scenarios?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's needed if the class being instrumented is in the method signature of the advice. I think Protocol is being instrumented, so I don't think this is needed.

@tylerbenson tylerbenson changed the title Redis support via Jedis 2.9 client Redis support via Jedis client Feb 23, 2018
@tylerbenson
Copy link
Copy Markdown
Contributor

Lets get an approval from @realark then we can merge.

@tylerbenson tylerbenson requested a review from realark February 23, 2018 01:43
@tylerbenson tylerbenson force-pushed the jedis branch 2 times, most recently from fe3721d to 1064922 Compare February 23, 2018 02:18
Also change instrumentation binding for better coverage.
@tylerbenson tylerbenson merged commit 72481c1 into DataDog:master Feb 23, 2018
@n0mer
Copy link
Copy Markdown

n0mer commented Mar 2, 2018

@tylerbenson @gihad is Lettuce integration planned?

spring-data-redis 2.0 supports reactive redis: https://docs.spring.io/spring-data/redis/docs/2.0.4.RELEASE/reference/html/#redis:reactive

@tylerbenson
Copy link
Copy Markdown
Contributor

I’m not familiar with lettuce. I’d suggest contacting support and submit a feature request with both of those.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: others All other instrumentations tag: community Community contribution

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants